Ranking Decision-Making Criteria for Early Adoption of Innovative Surgical Technologies

This decision analytical modeling study investigates the weighting of criteria used by surgeons, surgical administrators, and health technology assessment experts to make decisions concerning the early adoption of innovative surgical technologies.


Introduction
Continuous innovation in surgery is crucial in improving patient outcomes, minimizing complications, and allowing more effective and efficient surgical care. 1 The process of technology adoption over time is typically illustrated as a classical bell curve distribution, with the first group of people to use a new product called innovators, followed by early adopters, the early majority, and the late majority; the last group consists of laggards. 2While surgeons are generally risk averse and favor the status quo, there are surgeons who are early adopters of new technologies. 3These surgeons are more likely to be opinion leaders, closely watch for new innovations, embrace technological innovation, and be ready to adopt new technologies early. 4e decision-making process to purchase and adopt new surgical technology is complex and multifaceted, involving multiple stakeholders.It is not standardized and varies by country and health care system.In the US, after a technology is approved by the Food and Drug Administration, technology adoption is decided by the hospital administration and surgical departments. 5,6In the United Kingdom, after a new technology receives approval by the Medicines and Healthcare Regulatory Authority, 6 the National Institute for Health and Care Excellence conducts health technology assessments (HTAs) and provides recommendations for technology adoption to the National Health Service based on a cost-effectiveness analysis. 7In Canada, the decision to adopt and reimburse medical devices is decentralized in the publicly funded health care system. 8The Health Canada Medical Devices Directorate regulates medical devices for human use. 9While HTA is not mandatory in Canada, the Canadian Agency for Drugs and Technologies in Health provides information to some provincial HTAs, while other provinces have their own provincial HTA agencies and bodies, such as in Quebec (Institut National d'Excellence en Santé et en Services Sociaux) and Ontario (Health Quality Ontario, Evidence Development and Standards Division). 10In these provinces, technologies are assessed based on requests from hospital administrators and physicians. 10 using evidence-based criteria, decision-makers can evaluate the potential impact of a new surgical technology, weigh disadvantages, and make an informed decision that aligns with organization goals, needs, and resources.2][13] These include criteria specifically related to the technology, surgeon, patients being treated, hospital, and health care system. 14,15For example, in the highly competitive US health care system, 1 important criterion is the potential benefit for an institution to promote itself as cutting edge and a center of excellence and, thus, accept a higher volume of patients. 16In Canada, other criteria include disease burden, equity of access, and feasibility of implementation. 10e weighting of criteria is an essential step in decision-making because it simplifies the process by prioritizing the most relevant factors and allows for a more objective decision that aligns with organization goals and priorities.Organizations may have their own unique criteria that are specific to their needs and circumstances.Currently, there is no decision-making framework for the earlyadoption stage of novel surgical technologies.The aim of this study was to investigate the relevant weighted criteria that decision-makers, including surgeons, HTA experts, and surgical administrators, may use to make informed decisions for the early adoption of innovative surgical technologies.

Methods
No ethical or institutional review board approval was needed for this decision analytical modeling study because no medical devices, drugs, or patients were involved in the study; the McGill University Health Centre institutional review board does not require approval for public health surveillance activities, as defined by the Quebec Ministry as evaluation of care and services.
Participants provided consent to take part in the study.The Consolidated Health Economic Evaluation Reporting Standards (CHEERS) reporting guideline was adopted and modified to fit this multi-criteria decision analysis (MCDA) study by replacing the time horizon and economic aspect with using the analytical hierarchy process (AHP) and weighting of criteria as per Professional Society for Health Economics and Outcomes Research (ISPOR) guidance.

Study Design and Setting
Surgical department chairs, divisional chiefs, and procurement directors from the major Canadian university teaching hospitals, as well as members of all Canadian provincial HTA committees, were identified from their respective websites.Each was sent an invitation by email explaining the project and requesting participation in the study.An attempt was made to recruit individuals from all provinces and from diverse surgical disciplines.In the first round of surveys, 12 experts were questioned only to confirm criteria in the published literature or provide additional criteria.Experts were from McGill University and all divisions of surgery and included directors of all departments responsible for procurement at the McGill University teaching hospital.The MCDA study then used these literature-based criteria and the few additional criteria identified by experts.Decision criteria were prioritized and ranked using MCDA in accordance with the ISPOR. 17MCDA considers multidimensional factors and enables comparison in 1 overall appraisal.Participants surveyed for this study included 3 main stakeholder groups: surgeons, hospital administrators involved in the decision-making regarding and procurement of surgical devices, and provincial HTA committee members.All participants were experts in their fields, worked in Canada, and were identified by study investigators and by contacting HTA agencies across Canada (Table 1).After experts consented to participate in the study through email invitations, they were sent questionnaires electronically, and responses were collected anonymously.

Selection of Decision-Making Criteria Categories
We identified 33 decision-making subcriteria within 7 domains from a previously published study. 18 the first round of surveys, 12 experts were sent a questionnaire to evaluate the pertinence of these 33 subcriteria and determine if further subcriteria needed to be added.All newly identified subcriteria were then added to 1 of the 7 domains with the corresponding theme.Final decisionmaking domains and subcriteria were evaluated in the second step of the analysis.These experts were not part of the next round of surveys.

Prioritization and Ranking of Domains
In the second round of surveys, 33 of 45 experts agreed to participate in the study (response rate, 73.3%).Experts included 16 surgeons, 6 hospital administrators, and 11 provincial HTA committee members.Experts were sent a questionnaire that had 2 main objectives: first, to rank the importance of each subcriteria and second, to determine which domain was more important.We determined that the meaning of each domain in the questionnaire was clear to experts because each criterion was listed with its subcriteria, no respondents expressed uncertainty with a criterion, and no experts left a criterion question unanswered because they were not clear on its meaning.

Data Analysis and Model Development
Data were analyzed using a mixed-model methodology that includes domain prioritization by weighting using the AHP model and subcriteria ranking using the direct elicitation method, the Likert scale.To obtain an importance ranking, we asked respondents to consider each subcriterion independently and rank them on a 5-point Likert scale, with 1 characterizing an irrelevant criterion and 5 characterizing an absolutely relevant criterion (eAppendix in Supplement 1).Responses from the 33 experts were then analyzed using the arithmetic mean, and subcriteria were reordered based on their rankings from most to least important.
Second, domains were prioritized using the compositional pairwise comparison matrix, the AHP model (eAppendix in Supplement 1).0][21] The geometric mean was then estimated to combine individual pairwisecomparison judgments of the 33 experts into a pairwise-comparison matrix.[21]

Statistical Analysis
To verify the consistency of comparisons, the consistency index was applied for the entire pairwisecomparison matrix. 20,21The consistency ratio was then used to determine whether the matrix was sufficiently consistent with a random index (equaling 1.32) for the 7 categories. 20,21A consistency ratio less than 0.10 indicated that there was consistency within expert responses and the matrix was sufficient (eTable 2 in Supplement 1). 12 experts with a PhD degree.In the first questionnaire, experts validated and confirmed that the 7 domains and 33 subcriteria were pertinent.In addition, experts identified 11 new subcriteria that were important and were grouped with the 33 subcriteria identified in the literature. 18These additional subcriteria were cost-effectiveness, depreciation cost, certification of technology, percentage of use, maintenance availability, availability of evidence, alternatives available, being an academic center of excellence, evidence of peer-reviewed publications, and disease burden.

Analysis of Domains
Table 2 shows the full model and analysis results.Of 7 domains, clinical outcome was determined to be the most important domain, accounting for 42.9% (priority vector, 0.429) of the decisionmaking process.This domain included subcriteria of safety, efficacy, effectiveness, prevention of adverse effects, evidence of peer-reviewed assessments, and disease burden.This was 3 times more important than the next most important domain, patients and public relevance (priority vector, 0.135).Remaining domains each contributed less than 10% (Figure 1).Priority vectors were 0.099 for hospital-specific criteria, 0.092 for technology-specific criteria, 0.087 for physician-specific criteria, 0.083 for economic criteria, and 0.075 for policies and procedures.
In a subanalysis for surgeons only, the clinical outcomes domain was again found to have the highest priority vector, at 0.425.This was again followed by patients and public relevance, with a priority vector of 0.12.Next, was physician-specific criteria (priority vector, 0.113), technologyspecific criteria, (priority vector, 0.112), and hospital-specific criteria (priority vector, 0.107).The lowest priority vectors were again found with policies and procedures (0.066) and finally economic domains (0.059).
In a subanalysis including nonsurgeons only, the clinical outcomes domain was again found to have the highest priority vector, at 0.242, followed by patients and public relevance, with a priority vector of 0.184.This was followed by policies and procedures (priority vector, 0.135), economicspecific criteria (priority vector, 0.133), and technology-specific criteria (priority vector, 0.121) domains.The lowest priority vectors were hospital-specific criteria (0.110) and finally physicianspecific criteria (0.075).There were no statistically significant differences in priority weights in the 7 domains from the 3 groups.
The subanalysis for non-surgeons identified that safety and effectiveness were equally the most important subcriteria (priority vector, 4.938), followed by efficacy, feasibility of implementation, and structural management equally at a priority vector of 4.813.The lowest-scoring subcriteria were political (priority vector, 2.75) and publicity and awareness and appeals (priority vector, 2.688).
The consistency ratio of comparisons was found to be 0.006 (<0.1), indicating that there was consistency within expert responses and that the matrix was sufficient.In addition, the statistical analysis using ANOVA showed that there was no significant difference in priority vector weights among responses from all expert groups, suggesting that priority vectors from all expert responses

Discussion
This decision analytical modeling study identified clinical outcomes as the most important domain in decision-making regarding early adoption of innovative surgical technologies, accounting for 42.9% of the decision-making process.Safety, effectiveness, and efficacy were the most important individual subcriteria.This study's finding that clinical outcomes were of principal importance aligns with the rationale of adopting technologies to improve patient care.Patient safety and effectiveness were considered by surgeons and nonsurgeons to be the most important factors in adopting new technologies [13][14][15][16][17][18][19][20][21][22][23]  a Prioritization is from the analytical hierarchy process model.
b Domain prioritization by weighting used the analytical hierarchy process model, and subcriteria ranking used the direct elicitation method, the Likert scale.given that clinical outcomes are the mainstay for determining health-related quality of life and life expectancy. 22,24However, previous studies did not consider the technology adoption life cycle.In the early-adoption phase, the extent of clinical outcomes information is limited and may comprise only a few publications and conference presentations.This does not negate the importance of clinical outcomes but suggests that the expectation of having definitive scientific confirmation is unrealizable, clinical outcome data will be short term in nature, and other sources of data need to be used.These data include all available data from premarket and postmarket phases, such as postmarket surveillance reports, registries containing unpublished clinical performance or effectiveness data, adverse events held by the manufacturer or regulatory authorities, and any clinically relevant field corrective actions, such as recalls, notifications, and hazard alerts.
Furthermore, when assessing a new surgical technology, it is critical that the current standard of care be considered and used as the standard of comparison.In addition, given the paucity of data, the possible adoption of the new technology needs to be considered in the context of the currently available technology.
Results from this study have several policy implications.First, the agreement between surgeons and nonsurgeons suggests that the decision-making criteria can be harmonized.Surgeons are the primary users of surgical technologies and have a deep understanding of the technical aspects of the procedure and needs of the patient.Nonsurgeons, such as administrators, have a broader    perspective on the financial and organizational aspects of the technology adoption. 25Agreement by these groups suggests that the criteria may be relevant and appropriate and that they consider the different perspectives and expertise of both groups. 4,11,12,14cond, our results suggest that HTA bodies should include representation of all stakeholders, including surgeons.Surgeons play a critical role in the identification of new technology in the earlyadoption phase given that clinical results may be primarily from conferences and individual communications rather than extensive published literature and so would be available later in the technology life cycle. 25This information can provide important insight into the indications and benefits of new technologies. 13,25,26ird, this study's findings suggest that not all criteria are or should be equally weighted in importance.Not all criteria were considered as critical given that they may have had different levels of impact on patient outcomes and the overall success of the technology.By assigning different weights to criteria, decision-makers can ensure that the most important factors are given the most consideration when making decisions about new surgical technology adoption.The importance of weighing each criterion is particularly relevant given that the cost of the technology is commonly felt to be a primary criterion in decision-making. 16,23,27urth, this study found that economic and policies and procedures domains were the least important and cost of the technology ranked 32 of 44 among subcriteria.The cost of technology should be considered in the long term rather than just considering upfront costs.A technology that may have a higher upfront cost but that is associated with better patient outcomes and fewer complications in the long term may ultimately be more cost-effective.Cost-effectiveness analyses tend to use endogenous costs of technologies (final costs for payers) in their analyses rather than exogenous costs, which reflect true production costs. 28This undermines the value of costeffectiveness analysis that relates to the best use of resources for maximal health benefits. 29,30The opportunity cost would be lower societal benefits for alternative resources without maximizing health outcomes. 29,30This implies that cost-effectiveness analysis would not be considered an optimal method alone in HTA to influence decisions on technology adoption.

Limitations
Although this study has been designed with a mixed model methodology, there are some limitations.
First, domains and subcriteria identified from the literature included in this study come from the limited number of studies found, which might not capture the full spectrum of relevant domains for all hospitals.However, the first step in this study was to validate domains in the literature and determine if other domains should be included.This resulted in 11 additional subcriteria added to the analysis.Second, adoption criteria for new surgical technology were determined by pairwisecomparison matrices, and expert responses may, in part, reflect the Canadian health care system in which they worked.This may affect the generalizability of subcriteria in other countries with different health care systems.Additionally, numerous questions can lead to fatigue and confound some findings.Thus, questions were asked in more than 1 way (pairwise and direct ranking) to ensure that responses were reproducible.Third, prioritization and ranking needs to be a dynamic process.
Domains and weights identified in this study reflect a static assessment of present surgical and hospital priorities and should be reassessed over time to stay current and accurately reflect the technology and patient needs.Fourth, more experts, representing all provinces and surgical disciplines and with a more equitable distribution of sexes would be needed for further criteria validation.Although it would be ideal if our cohort had been half male and half female, the 27% of respondents in our study who were female is not different than the distribution of surgeons in Canada.

JAMA Network Open | Health Policy
Ranking Decision-Making Criteria for Early Adoption of Innovative Surgical Technologies

Figure 3 . 2
Figure 3. Prioritization of 44 Subcriteria by Criteria Domain Safety Effectiveness Efficacy Prevention of adverse effects Evidence of peer-reviewed publications Disease burden

Table 2 .
Representation of Criteria Prioritization for Surgical Technology Adoption a (continued)